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Abstract — We  describe  how  a  virtual  node  abstraction  layer 
can  be  used  to  coordinate  the  motion  of  real  mobile  nodes 
in  a  region  of  2-space.  In  particular,  we  consider  how  nodes 
in  a  mobile  ad  hoc  network  can  arrange  themselves  along  a 
predetermined  curve  in  the  plane,  and  can  maintain  themselves 
in  such  a  configuration  in  the  presence  of  changes  in  the 
underlying  mobile  ad  hoc  network,  specifically,  when  nodes  may 
join  or  leave  the  system  or  may  fail.  Our  strategy  is  to  allow  the 
mobile  nodes  to  implement  a  virtual  layer  consisting  of  mobile 
client  nodes,  stationary  Virtual  Nodes  (VNs)  for  predetermined 
zones  in  the  plane,  and  local  broadcast  communication.  The 
VNs  coordinate  among  themselves  to  distribute  the  client  nodes 
between  zones  based  on  the  length  of  the  curve  through  those 
zones,  while  each  VN  directs  its  zone’s  local  client  nodes  to  move 
themselves  to  equally  spaced  locations  on  the  local  portion  of 
the  target  curve. 

Index  Terms — Motion  coordination,  virtual  nodes,  hybrid 
systems,  hybrid  I/O  automata. 

I.  Introduction 

Motion  coordination  is  the  general  problem  of  achiev¬ 
ing  some  global  spatial  pattern  of  movement  in  a  set  of 
autonomous  agents.  An  important  motivation  for  studying 
distributed  motion  coordination,  that  is,  coordination  among 
agents  with  only  local  communication  ability  and  therefore 
limited  knowledge  about  the  state  of  the  entire  system, 
stems  from  the  developments  in  the  field  of  mobile  sensor 
networks.  Previous  work  in  this  area  includes  different  co¬ 
ordination  goals,  for  example:  flocking  [9],  rendezvous  [1], 
[10],  [13],  deployment  [2],  pattern  formation  [15],  and  ag¬ 
gregation  [7],  Owing  to  the  intrinsic  decentralized  nature 
of  sensor  network  applications  like  surveillance,  search  and 
rescue,  monitoring,  and  exploration,  centralized  or  leader 
based  approaches  are  ruled  out.  However,  the  lack  of  central 
control  makes  the  programming  task  quite  difficult. 

In  prior  work  [3],  [5],  [6],  [4],  we  have  developed  a  notion 
of  “virtual  nodes”  for  mobile  ad  hoc  networks.  A  virtual 
node  is  an  abstract,  relatively  well-behaved  active  node  that 
is  implemented  using  less  well-behaved  real  nodes.  Virtual 
nodes  can  be  used  to  solve  problems  such  as  providing 
atomic  memory  [5],  geographic  routing  [3],  and  point-to- 
point  routing  [4]. 

In  this  paper,  we  explore  a  framework  for  using  virtual 
nodes  to  solve  motion  coordination  problems.  We  consider 
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virtual  nodes  associated  with  predetermined,  well-distributed 
locations  in  the  plane,  communicating  among  themselves  and 
with  mobile  “client  nodes”  using  local  broadcast.  Roughly 
speaking,  each  virtual  node  is  designated  a  certain  zone  in  the 
plane — disjoint  from  the  zones  of  the  other  virtual  nodes — 
and  is  emulated  by  all  the  mobile  nodes  that  are  present 
in  its  zone.  The  VN  abstraction  makes  programming  easier 
by  providing  a  centralized  controller  with  reliable  storage  (a 
virtual  node)  for  each  disjoint  zone  in  the  plane. 

In  this  paper  we  describe  a  framework  for  using  vir¬ 
tual  nodes  to  solve  a  simple  motion  coordination  prob¬ 
lem,  namely,  uniformly  positioning  the  mobile  nodes  on 
a  known  differentiable  curve.  The  framework  can  be  used 
to  implement  distributed  algorithms  for  more  complicated 
motion  coordination  problems  as  well.  Using  the  virtual 
node  abstraction  for  solving  a  coordination  problem  reduces 
to  achieving  coordination  among  stationary  centralized  con¬ 
trollers  with  a  priori  known  locations.  Further,  writing  the 
coordination  algorithm  for  the  virtual  nodes  amounts  to 
“plugging  in”  control  functions  in  the  virtual  node  programs. 

In  addition  to  describing  the  framework  for  using  virtual 
nodes  to  solve  coordination  problems,  we  alsobriefly  describe 
one  way  of  implementing  virtual  nodes  using  the  real  mobile 
nodes.  We  use  the  Hybrid  I/O  Automata  (HIOA)  mathemat¬ 
ical  framework  [12]  for  describing  the  components  in  our 
systems. 

The  paper  is  organized  as  follows:  Section  II  describes  the 
underlying  mobile  network.  Section  III  describes  our  virtual 
node  layer.  Section  IV  defines  the  motion  coordination 
problem  we  consider.  Section  V  describes  an  algorithm  for 
solving  this  motion  coordination  problem  using  the  virtual 
node  layer.  Section  VI  gives  the  proofs  of  correctness  of  the 
algorithm.  Section  VII  presents  simulation  results  for  our 
algorithm.  Section  VIII  outlines  one  way  to  implement  the 
virtual  node  layer,  and  Section  IX  concludes. 

II.  The  Physical  Layer 

Our  physical  model  of  the  system  consists  of  a  finite 
but  unknown  number  of  communicating  physical  nodes  in 
a  bounded  square  B  in  R2.  We  assume  that  each  node  has  a 
unique  identifier  from  a  set  I.  Formally,  our  physical  layer 
model  consists  of  three  types  of  HIOA  (see  Figure  1):  (1) 
automata  PN ,  to  model  physical  nodes  with  identifiers  i  £ 
I ,  (2)  a  LBcast  automaton  that  models  the  local  broadcast 
communication  service  between  the  physical  nodes,  and  (3)  a 


“real  world”  automaton  RW  to  model  the  physical  locations 
of  all  the  nodes  and  the  real  time. 


A 


Fig.  1.  The  Physical  Layer:  PN  automata  communicate  with  each  other 
through  an  LB  cast  service  and  receive  time  and  location  information 
continuously  from  RW. 


Figure  2  shows  the  required  components  of  each  automa¬ 
ton  PN,\  it  may  have  other  internal  variables  (initially  set 
to  unique  initial  values)  and  actions,  which  are  not  specified 
here.  PN i  continuously  receives  from  RW  the  current  time 
as  the  input  variable  realtime  and  its  position  as  the  input 
variable  x,,  and  communicates  its  velocity  to  RW  through 
the  output  variable  v,  .  The  speed  of  PN ;  is  bounded  by  vc. 
The  trajectories  of  the  continuous  variable  V;  and  the  effects 
of  the  send  and  receive  actions  are  unspecified.  At  each 
point  PN is  either  in  active  or  inactive  mode;  we  assume 
that,  initially,  finitely  many  nodes  are  active.  The  fail*  input 
action  sets  the  mode  to  inactive  and  all  internal  variables 
to  their  initial  values,  and  the  recover;  input  action  sets  the 
mode  to  active.  In  inactive  mode,  all  internal  and  output 
actions  are  disabled,  no  input  action  except  recover;  affects 
the  internal  or  output  variables,  and  during  trajectories,  the 
locally-controlled  variables  remain  constant  and  the  velocity 
v;  remains  zero.  Thus,  we  assume  that,  in  inactive  mode, 
PN i  stops  moving.  We  model  the  departure  of  a  node  from 
B  as  a  failure.  For  convenience,  we  assume  that  transitions 
are  instantaneous. 

The  PNs  communicate  using  a  local  broadcast  service, 
LBcast,  which  is  a  generic  local  broadcast  service  param¬ 
eterized  by  a  radius  Rp  and  a  maximum  message  delay 
dp.  The  LBcast(Rp,dp)  service  guarantees  that  when  PW; 
performs  a  send(m);  action  at  some  time  t,  the  message 
is  delivered  within  the  interval  [t,  t  +  dp\,  by  a  receive(m)j 
action,  to  every  PN j  that  remains  in  active  mode  and  within 
Rp  distance  of  PN ;  for  the  entire  interval  [t,t  +  dp\. 

The  RW  automaton  (see  Figure  3)  erves  to  model  realtime 
and  locations  of  all  the  nodes.  It  reads  the  velocity  output  v; 
from  each  PN ;,  i  £  X,  and  computes  the  position  X;  from 
the  velocity  V;  for  PW;  and  the  LBcast  automaton.  LBcast 
requires  the  node  position  information  because  it  guarantees 
delivery  only  between  “nearby”  nodes.  RW  also  produces 


Signature: 

Input 

receive(m); 

fail, 

recover, 

Output 

send(m); 

Variables: 

Input 

X;  e  B 

realtime  G  R—° 

Output 

V;  G  if2,  j  V;  |  <  Vc 

Internal 

mode  g  (active,  inactive} 
Finite  set  of  other  variables. 


Transitions: 

Input  fail; 

Effect 

V  ;  <-  0 

mode  4—  inactive 

Other  internal  variables  <—  initial 

Input  recover; 

Effect 

mode  <—  active 


initially  set  to  unique  initial  values. 


Fig.  2.  Hybrid  I/O  Automaton  PN ;. 


Variables: 

Input 

v;  G  if2,  for  each  i  G  X 

Output 

x;  G  B.  for  each  i  G  I 
realtime  G  if-0 

Internal 

x;  G  B.  for  each  i  G  X,  initially  arbitrary 
clock  G  if-0,  initially  0 

Trajectories: 

Invariant 

x;  G  B.  for  each  i  Gl 

Evolve 

x;  =  x;,  for  each  i  G  X 
d(x;)  =  v;,  for  each  iGl 
realtime  =  clock 
d(clock)  =  1 


Fig.  3.  RW  automaton. 


realtime  for  all  physical  layer  components. 

III.  The  Virtual  Node  Layer 

The  bounded  square  B  is  partitioned  into  a  finite  set  of 
zones  Bh,  h  £  H.  For  simplicity  we  assume  B  is  a  m  x  m 
square  grid,  with  each  grid  square  corresponding  to  a  zone 
and  having  sides  of  length  b.  Each  boundary  point  of  a  square 
is  unambiguously  assigned  to  one  zone.  The  index  set  H  is 
the  set  of  coordinates  of  the  centers  of  all  squares.  For  each 
Bh,  the  set  NbrSh  contains  the  zone  identifiers  of  the  north, 
south,  east,  and  west  neighboring  grid  squares. 

Our  virtual  layer  abstraction  (see  Figure  4)  consists  of: 
(1)  client  node  automata  GW;  with  identifiers  i  £  X,  (2) 
one  stationary  virtual  node  automaton  VN  h  for  each  h  £ 
TL,  located  at  the  center  o/,  of  the  square  Bh,  (3)  a  virtual 
communication  service,  VLBcast  =  LBcast{Rv,  dv),  for  the 
VNs  and  the  CNs,  and  (4)  an  automaton  RW  to  model  the 
physical  locations  of  all  the  CN s  and  the  real  time. 

A  client  node  automaton  GW,,  i  £  X,  is  a  portion  of  a 
PN i  automaton  that  has  the  input  variables  realtime  and 
X;  from  the  RW  automaton  and  an  output  variable  V;  to  the 


IV.  The  Motion  Coordination  Problem 


Fig.  4.  Virtual  Node  Layer:  VN s  and  CN s  communicate  using  the 
VLBcast  service. 


RW  automaton.  With  respect  to  failures,  an  automaton  CN, 
behaves  the  same  as  PN i.  CN i  also  has  send  and  receive 
actions  for  interacting  with  the  VLBcast  service. 

A  virtual  node  automaton  VN  h,  h  £  ft,  is  an  MMT 
automaton  [11],  [14]  parameterized  by  a  time  upper  bound, 
dMMT ;  it  has  no  realtime  clock  variable.  MMT  automata  are 
discrete  I/O  automata  that  have  a  “task”  structure,  which  is  an 
equivalence  relation  on  the  set  of  locally-controlled  actions, 
such  that  from  a  point  in  an  execution  where  a  task  becomes 
enabled,  within  at  most  time  dMMT,  some  action  in  that 
task  must  occur.  VNf,  can  experience  a  fail ?,  input,  disabling 
internal  and  output  actions,  preventing  any  inputs  other  than 
recover?,  from  resulting  in  state  changes,  and  setting  the 
automaton  to  an  initial  state.  If  a  recover?,  occurs  at  a  failed 
VN,  the  VN  actions  become  enabled  with  all  tasks  restarted. 
If  VN h  is  failed  and  a  CN  is  in  If,  and  remains  active  in 
the  zone  for  dr  time,  then  a  recover?,  occurs  within  that  d, 
time.  VN f,  communicates  with  other  VNs  and  CN s  using 
the  VLBcast  service  through  send?,  and  receive?,  actions. 

VLBcast  is  an  LBcast  service  (as  described  in  the  phys¬ 
ical  layer)  for  the  virtual  layer,  parameterized  by  radius  Rv 
and  maximum  message  delay  dv,  where  Rv  >  b.  It  allows 
VN h  to  communicate  with  each  VN g  such  that  g  £  Nbrsh, 
and  with  CN s  that  are  located  in  If,.  It  does  not  allow  CN 
automata  to  communicate  with  one  another. 

The  RW  automaton  in  the  virtual  layer  is  similar  to  the 
one  in  the  physical  layer,  but  here  it  communicates  (through 
the  realtime  and  x  variables)  only  with  the  CN  automata 
and  the  VLBcast  automaton,  and  not  the  VN  automata. 

This  virtual  layer  will  be  used  in  Section  V  to  implement 
a  solution  to  the  distributed  motion  coordination  problem. 
Details  of  how  this  virtual  layer  can  be  implemented  using 
the  physical  layer  are  in  Section  VIII.  There  we  further 
discuss  the  relation  between  the  parameters  dMMT ,  dr,  dv, 
and  Rp ,  the  physical  layer  broadcast  radius. 


A  differentiable  parameterized  curve  T  is  a  differentiable 
map  P  — ►  B,  where  the  domain  set  P  of  parameter  values  is 
an  interval  in  the  real  line.  The  curve  T  is  regular  if  for  every 
p  £  P,  |r,(p)|  ^  0.  For  a,b  £  P,  the  arc  length  of  a  regular 
curve  T  from  a  to  b,  is  given  by  s(T,a,b)  =  f  *'  |r'(p)|dp. 
r  is  said  to  be  parameterized  by  arc  length  if  for  every 
p  £  P,  |r'(p)|  =  1.  For  a  curve  parameterized  by  arc  length, 
s(r,  a,  b)  =  b  —  a. 

For  a  given  point  x  £  B,  if  there  exists  p  £  P  such  that 
r(p)  =  x,  then  we  say  that  the  point  x  is  on  the  curve  F; 
abusing  the  notation,  we  write  this  as  x  £  F.  We  say  that  T  is 
a  simple  curve  provided  for  every  x  £  T,  r-1(x)  is  unique. 
A  sequence  xi, . . .  ,xn  of  points  in  B  are  said  to  be  evenly 
spaced  on  a  curve  T  if  there  exists  a  sequence  of  parameter 
values  pi  <  p2  ■  ■  ■  <  pn,  such  that  for  each  i,  1  <  i  <  n, 
F (pi)  =  Xj,  and  for  each  i,  1  <  i  <  n,  pi—pi- 1  =  pi+i—p,. 

In  this  paper  we  fix  F  to  be  a  simple,  differentiable  curve 
that  is  parameterized  by  arc  length.  Let  Ph  =  {p  £  P  : 
r(p)  £  Bh }  be  the  domain  of  F  in  zone  If,  C  B.  The  local 
part  of  the  curve  T  in  zone  If,  is  the  restriction  F?,  :  P?,  — > 
If,.  We  assume  that  Ph  is  convex  for  every  zone  If,  C  B\  it 
may  be  empty  for  some  Bh-  We  write  \Ph  \  for  the  length  of 
the  curve  T?,.  We  define  the  quantization  of  a  real  number 
x  with  quantization  constant  a  >  0  as  qa{x)  =  [-]<r.  For 
the  remainder  of  the  paper  we  fix  a  and  write  (/?,  as  an 
abbreviation  for  qa  ( |  P?,  | ) .  We  write  qrrnr,  for  the  minimum 
nonzero  qh,  and  qmax  for  the  maximum  qh- 

Our  goal  is  to  design  an  algorithm  that  runs  on  the  physical 
nodes  such  that,  if  there  are  no  failures  or  recoveries  of 
physical  nodes  after  a  certain  point  in  time,  then:  (1)  within 
finite  time  the  set  of  nodes  in  each  zone  Bh,  h  £  ft,  becomes 
fixed,  and  the  size  of  the  set  is  “approximately”  proportional 
to  the  quantized  length  qh,  (2)  within  finite  time  all  physical 
nodes  in  Bh  for  which  qh  /  0  are  located  on  I’?,,  and  (3)  in 
the  limit  all  the  nodes  in  each  If,  are  evenly  spaced  on  F?,. 

V.  Solution  Using  Virtual  Node  Layer 

The  Virtual  Node  abstraction  is  used  as  a  means  to 
coordinate  the  movement  of  client  nodes  in  a  zone.  A  VN 
controls  the  motion  of  the  CN  s  in  its  zone  by  setting  and 
broadcasting  target  waypoints  for  the  CN s:  VN h,  h  £  ft, 
periodically  receives  information  from  clients  in  its  zone, 
exchanges  information  with  its  neighbors,  and  sends  out  a 
message  containing  a  calculated  target  point  for  each  client 
node  “assigned”  to  zone  Bh-  Informally,  VN ?,  performs  two 
tasks  when  setting  the  target  points:  (1)  it  re-assigns  some  of 
the  CN s  that  are  assigned  to  itself  to  neighboring  VN s,  and 
(2)  it  sends  a  target  position  on  F  to  each  CN  that  is  assigned 
to  itself.  The  objective  of  (1)  is  to  prevent  neighboring  VNs 
from  getting  depleted  of  CN s  and  to  achieve  a  distribution  of 
CN s  over  the  zones  that  is  proportional  to  the  length  of  F  in 
each  zone.  The  objective  of  (2)  is  to  space  the  nodes  evenly 
on  r  within  each  zone.  A  CN,  in  turn,  receives  its  current 
position  information  from  RW  and  its  target  location  from  a 
VN,  and  continuously  computes  a  velocity  vector  that  will 
take  it  to  its  latest  received  target  point. 


In  our  algorithm  each  virtual  node  VN  /,  uses  only  in¬ 
formation  about  the  portions  of  the  target  curve  F  in  zone 
Bh  and  the  neighboring  zones.  For  convenience,  we  assume 
that  all  client  nodes  know  the  complete  curve  F;  we  could 
instead  model  the  client  nodes  in  /!/,  as  receiving  inputs  from 
another  automaton  about  the  nature  of  the  curve  in  zone  B /> 
and  neighboring  zones  only. 

A.  Client  Node  Algorithm 

The  algorithm  for  the  client  node  CN  ( S)i ,  i  Si,  appears 
in  Figure  5.  The  client  node  follows  a  round  structure,  where 
rounds  begin  at  times  that  are  multiples  of  S.  Recall  that 
VN  automata  do  not  have  access  to  realtime  whereas  CN 
automata  do.  To  help  VN s  follow  the  round  structure,  the 
CN s  send  “trigger”  messages  to  prompt  the  VN s  to  perform 
transitions. 

At  the  beginning  of  each  round,  a  CN  sends  a  cn-update 
message  to  its  local  VN  (that  is,  the  VN  in  whose  zone 
the  CN  currently  resides).  The  cn-update  message  tells  the 
local  VN  the  CN' s  id,  its  current  location  in  B,  and  current 
round  number. 

The  CN  then  sends  an  exchange-trigger  message  d„  +  e 
later  to  its  local  VN.  An  additional  d m mt  +  2 dv  +  e  time 
later,  the  CN  sends  a  target-trigger  message  to  its  local 
VN.  Both  these  messages  are  trigger  messages  that  include 
the  CN' s  current  location  and  the  current  round  number, 
used  by  the  local  VN  to  determine  whether  the  CN  is  in  its 
zone  and  what  the  current  round  number  is. 

CN i  processes  only  one  kind  of  message,  target-update 
messages  sent  by  its  assigned  VN.  Each  such  message 
describes  the  new  target  location  x*  for  CN  t,  and  possibly  an 
assignment  to  a  different  VN.  CN *  continuously  computes 
its  velocity  vector  v,;,  based  on  its  current  position  x,  and  its 
target  position  x*,  as  v,:  =  i)c(xj  —  x*)/||x,;  —  x*||,  moving 
it  with  maximum  velocity  towards  the  target. 

B.  Round  structure 

The  VNh,  h  €  H,  algorithm  follows  the  CNs'  round 
structure.  However,  VN s  do  not  have  access  to  the  realtime 
variable  and  must  instead  rely  on  trigger  messages  from 
CN s  to  determine  when  enough  time  has  elapsed  to  perform 
required  actions.  Here  we  explain  how  we  implement  the 
round  structure  for  a  VN. 

Recall  that  at  the  beginning  of  a  round,  each  CN  sends 
a  cn-update  message  to  its  local  VN.  The  CNs  then  send 
exchange-trigger  messages  dv  +  e  after  the  beginning  of 
the  round,  enough  time  that  the  cn-update  messages  have 
already  been  delivered,  signaling  to  the  VN  that  it  has 
received  all  cn-update  messages  that  were  transmitted  at 
the  beginning  of  the  round  in  its  zone.  The  VN  waits  before 
using  information  from  the  cn-update  messages  until  it 
receives  one  of  the  CNs '  exchange-trigger  messages.  The 
VN  then  sends  vn-update  messages  to  its  neighbors. 

Each  CN  sends  a  target-trigger  message  to  its  local 
VN  an  additional  d\iMT  +  2 dv  +  e  time  after  it  sends  an 
exchange-trigger  message.  This  additional  time  is  enough 
for  all  the  following  to  have  happened:  (1)  each  neighboring 


Signature: 

Input 

receive(m)i,  m  g  ({target-update}  x  B) 

Output 

send(m)it  m  g  ({cn-update}  x  X  x  B  x  N) 

U  ({exchange-trigger,  target-trigger}  x  B  x  N) 

Internal 

inih 

Variables: 

Input 

Xi  G  B 

realtime  G  R—° 

Output 

Vi  G  R2 ,  velocity  vector 

Internal 

x*  GSU{1},  target  point,  initially  _L 

round,  next-exch,  next-target  G  N  U  {-L},  initially  _L 

Transitions: 

Internal  initi 
Precondition 

round  =  ± 

Effect 

round,  next-exch,  next -target  <—  \realtime/S~\ 
x*  <—  Xi 

Input  receive((target-update,  target))i 
Effect 

if  target(i)  V  null  then 
x*  <—  target(i) 

Output  send( (cn-update.  /,  Xi,  round))i 
Precondition 

realtime  =  round  ■  8 

Effect 

round  <—  round  +  1 

Output  send( (exchange-trigger,  xi,  next-exch))i 

Precondition 

realtime  =  next-exch  ■  5  +  dv  +  e 

Effect 

next-exch  <—  next-exch  +  1 

Output  send( (target -trigger,  x;,  next -target)) i 

Precondition 

realtime  =  next-target  ■  6  +  djvfMT  +  3 dv  +  2e 

Effect 

next-target  <—  next-target  +  1 

Trajectories: 

Evolve 

if  (x;  =  x*  or  x*  =  _L)  then  Vi  =  0 
else  Vi  =  vc  ■  (x*  —  Xi)/|jx*  —  x|| 

Stop  when 

round  =  ±  or  realtime  =  round  ■  8 

or  next-exch-5  +  dv  +  e  or  next-target-8  +  dMMT  +  3 dv  +  2e 


Fig.  5.  Client  node  CN(8)i  automaton. 


VN  has  received  an  exchange-trigger  message  from  a 
CN  in  its  zone  ( dv  time),  (2)  each  neighboring  VN  has 
performed  a  vn-update  transmission  to  its  neighboring  VN s, 
including  this  one  Mm mt  time),  and  (3)  the  neighboring 
VN  vn-update  messages  have  arrived  (dv  time).  When  a 
VN  first  receives  a  target-trigger  message  for  a  particular 
round  from  any  CN  in  its  region,  it  knows  it  has  received  any 
vn-update  messages  from  neighboring  VNs  for  the  round. 
The  VN  then  performs  some  computation  and  transmits  a 
target-update  message  to  CN s  local  to  it. 

A  target-update  message  might  not  be  received  by  a  CN 
until  (Im mt  +  2dv  time  after  the  CN  sent  the  target-trigger 
message.  This  accounts  for:  (1)  the  time  it  can  take  for  the 
target-trigger  message  to  be  received  by  the  VN  ( dv ),  (2) 
the  time  it  can  take  for  the  VN  to  perform  the  target-update 
broadcast  (< Immt ),  and  (3)  the  time  for  the  broadcast  to 
be  delivered  at  the  CN  ( dv ).  Given  the  maximum  distance 
between  a  point  in  one  zone  and  the  center  of  a  neighboring 
zone,  v/2.5 b  =  \/(3b/2)2  +  Jb] 2)2,  and  a  constant  speed  of 
vc  for  each  client  node,  it  can  take  up  to  v^p b  time  for  the 
CN  to  reach  its  target.  Also,  after  the  CN  just  arrives  in  the 
zone  it  was  assigned  to,  up  to  \/l0&/3  =  V2.5 b  •  |  distance 
from  where  it  started,  it  could  find  that  the  local  VN  is  failed, 
in  which  case  it  could  take  up  to  the  dr  VW-startup  time  for 
the  VN  to  recover. 

To  ensure  a  round  is  long  enough  for  a  client  node  to 
send  the  cn-update,  exchange-trigger,  and  target-trigger 

messages,  receive  a  target-update  message,  arrive  at  its  new 
assigned  target  location,  and  be  sure  a  virtual  node  is  alive  in 
its  zone  before  a  new  round  begins,  we  require  that  8  satisfy 

5  >  2 (Immt  +  5 dv  +  2e  +  max(V2.5b/vc,  s/l0b/3vc  +  dr). 

C.  VN  algorithm 

The  algorithm  for  virtual  node  VN(e,  pi,  p2)h>  h  £  H, 
appears  in  Figure  6,  where  e  €  Z+  and  pi,p2  €  (0,1) 
are  parameters  of  the  automaton.  VN h  collects  cn-update 
messages  sent  at  the  beginning  of  the  round  from  CN s 
located  in  its  zone,  aggregating  the  location  and  round 
information  from  the  message  in  a  table,  M.  When  VN  /, 
first  receives  an  exchange-trigger  message  for  a  particular 
round  from  any  CN  in  its  zone,  VN  /,  tallies  and  computes 
from  its  table  M  the  number  of  client  nodes  assigned  to  it 
that  it  has  heard  from  in  the  round,  and  sends  this  information 
in  a  vn-update  message  to  all  of  its  neighbors. 

When  VHh  receives  a  vn-update  message  from  a  neigh¬ 
boring  VN,  it  stores  the  CN  population  and  round  number 
information  from  the  message  in  a  table,  V.  When  VN h 
first  receives  a  target-trigger  message  for  a  particular  round 
from  any  CN  in  its  region,  VN  /,  uses  the  information  in  its 
tables  M  and  V  about  the  number  of  CN  s  in  its  zone  and  its 
neighbors’  zones  to  calculate  how  many  of  the  CN s  assigned 
to  itself  should  be  reassigned  and  to  which  neighboring  VNs. 
This  is  done  through  the  assign  function  (see  Figure  7) 
which  calculates  a  partial  function  assign  mapping  CN 
identifiers  to  zones  that  they  are  assigned  to.  If  the  number 
of  CN s  y(h)  assigned  to  VN h  exceeds  the  minimum  critical 


Signature: 

Input 

receive(m)fe,  m  e  ({cn-update}  x  1  x  B  x  N) 

U  ({exchange-trigger,  target-trigger)  x  B  x  N) 
U  ({vn-update}  x  H  x  N  x  N) 

Output 

send  (m)  h 

Constants: 

In  =  {{  €  Nbrs:  qg  ft  0} 

State  variables: 

M  :  I  — »  B  x  N,  partial  map  from  CN  ids  to  current  location  and 
round  number,  initially  0.  Accessors:  loc,  round. 

V  :  H  ->  N  X  N,  partial  map  from  VN  ids  to  the  number  of  CNs,  and 
round  number,  initially  {( g ,  (0.  0)}}  for  each  g  £  Nbrs  U  { h  } . 
Accessors:  num ,  round, 
send-buffer,  queue  of  messages,  initially  0. 
vn-done,  target-done  £  Z,  initially  0. 

Derived  variables: 

locM  =  A (i  £  id(M)).  loci M( /)) 
y  =  \(g  £  Nbrs  U  {It}).  nuin(V(g)) 

Transitions: 

Input  receive((cn-update,  id,  loc,  round))h 

Effect 

if  loc  £  B)  then 

M  ‘ —  M  U  {(id,  (loc,  round))} 

Input  receive((exchange-trigger,  loc,  round))h 
Effect 

if  (loc  £  Bit.  A  vn-done  f  round)  then 
for  each  i  £  id(M) 

if  round(M(i))  V  round  then 
M  <-  AT  \  {(/,  Af(i))} 

send-buffer  <—  send-buffer  U  {(vn-update.  It,  \M\,  round)} 
vn-done  <—  round 

Input  receive((vn-update,  id,  n,  round ))/, 

Effect 

if  id  £  Nbrs  then 
V(id)  <—  («,  round) 

Input  receive  ((target -trigger,  loc,  round))h 
Effect 

if  ( loc  £  Bf ,  A  target-done  f  round)  then 
Vih)  <—  (\M\,  round) 
for  each  g  £  Nbrs 

if  round(V(g))  f  round  then 

v(g)  -  (0,  0) 

let  target  =  calctarget(assign(/J(M),  y),  locM ) 

send-buffer  send-buffer  U  {(target-update,  target)} 
target-done  round 

Output  send(m)h 
Precondition 

send-buffer  /  0  A  m  —  head  (send -buffer) 

Effect 

send-buffer  tail  (send-buffer) 

Tasks  and  bounds: 

(send(ra)fr},  bounds  [0,  c^mmt] 


Fig.  6.  VN(e,  pi,  p2)h  IOA  signature,  variables,  transitions,  and  tasks, 
implementing  motion  coordination  algorithm  with  parameters:  safety  e,  and 
damping  pi,p2- 


Functions: 

function  assig r\(assignedM:  21,  y :  Nbrs  U{/i}  — ►  N):  X  — ►  Ti  = 
assign :  X  — ►  Tt,  initially  {(/,  h)}  for  each  i  G  assignedM 
n :  N,  initially  ?/(/i) 
ra:  N,  initially  0 
if  y(h)  >  e  then 
if  qh  j1  0  then 

let  lower  =  {g  G  /«:  ^pV{h)  >  J/(ff)} 

for  each  g  G  lower 

ra  <-  min([p2  •  [ff-J/M  -  3/(9)]/2(|fcwr|+l)J,  n  -  e) 
update  assign  by  reassigning  ra  nodes  from  h  to  g 
n  <—  n  —  ra 

else  if  In  =  0  then 

let  lower  =  {g  G  M?rs  :  y(/i)  >  2/(g)} 

for  each  g  G  /ow^r 

ra  <-  min([p2  •  [if(ft)  -  y{g)\/2(\lower\+\)\,  n  -  e) 
update  assign  by  reassigning  ra  nodes  from  h  to  g 
n  *—  n  —  ra 

else 

ra  <-  L(3/(^)  -  e)/NJ 

for  each  g  €  In 

update  assign  by  reassigning  ra  nodes  from  h  to  g 
return  assign 

function  calctarget (assign:  X  — >  Ti,  locM :  X  — >  Z3):  X  — >  H  = 
seq,  indexed  list  of  pairs  in  P  x  X,  initially  the  list,  for  each  iGX: 
assign(i)=  h  A  locM(i)  G  I\,  of  (p,  i)  where  /?=  r^1(/acM(r)), 
sorted  by  p,  then  i 
for  each  i  G  X  :  assign(i)  ^  aw// 
if  assign(i)  =  g  ^  h  then 
locM(i)  o5 
else  if  locM(i)  then 

locM(i)  <—  choose  {minxer h{dist(x,locM(i))}} 
else  let  p  =  (locM  (i)) ,  seq(k)  =  (p,  i) 

if  k  =  first(.veg)  then  locM(i)  <—  r^(inf(P^)) 
else  if  k  =  last  (seq)  then  locM(i)  r^(sup(P^)) 
else  let  seq(k  -  1)  =  (pk-i^k-i),  seq(k  +  1)  =  {pk+i^k+i) 
locM(i)  «-  Fh{p  +  P1  •  (Pfc-i+Pfc+i  _  p)) 
return  /ocM 


Fig.  7.  VN(e,  pi,  p2)h  IOA  functions. 


number  e,  then  the  assign  function  reassigns  some  of  the 
CNs  to  neighbors  of  VN  h- 

Let  Irih  denote  the  set  of  neighboring  VN s  of  VN p  that 
are  on  the  curve  T  and  yh(g ),  g  €  Nbrsh  U  {h},  denote 
the  number  num(Vh(g))  of  CNs  assigned  to  VN g.  If  ^ 
0,  meaning  VN g  is  on  the  curve  (lines  7-11),  then  we  let 
lowerh  denote  the  subset  of  Nbrsh  that  are  on  the  curve  and 
have  fewer  assigned  CNs  than  VN  /,  has  after  normalizing 
with  For  each  g  £  lowerh ,  VN p  reassigns  the  smaller 
of  the  following  two  quantities  of  CNs  to  VNg:  (1)  ra  = 
P2  ■  [^l lh(h)  -  yh(g)]/2(\lowerh\  +  1),  where  p2  <  1  is  a 
damping  factor,  and  (2)  the  remaining  number  of  CN s  over 
e  still  assigned  to  VN  h. 

If  qh  =  0,  meaning  VN g  is  not  on  the  curve,  and  VN h  has 
no  neighbors  on  the  curve  (lines  13-17),  then  we  let  lowerh 
denote  the  subset  of  Nbrsh  with  fewer  assigned  CNs  than 
VN h-  For  each  g  £  lowerh ,  VNg  reassigns  the  smaller  of 
the  following  two  quantities  of  CNs:  (1)  ra  =  p2  ■  [ yh(h )  — 
yh(g)]/2(\lowerh  |  +  1)  and  (2)  the  remaining  number  of 
CNs  over  e  still  assigned  to  VN h- 

VN h  is  on  a  boundary  if  qh  =  0,  but  there  is  a  g  £  Nbrsh 


with  qg  ^  0.  In  this  case,  ynih)  —  e  of  VN h's  CNs  are 
assigned  equally  to  neighbors  in  Inh  (lines  19-22). 

The  client  assignments  are  then  used  to  calculate  new 
target  points  for  local  CN s  through  the  calctarget  function 
(see  Figure  7).  This  function  assigns  to  every  CN i  assigned 
to  VN h  a  target  point  locMh(i)  £  Bg,g  £  Nbrsh  U  {h},  to 
move  to.  The  target  point  locMh(i)  is  computed  as  follows: 
If  CN i  is  assigned  to  VN g,  g  ^  h,  then  its  target  is  set 
to  the  center  og  of  Bg  (lines  30-31);  if  CN ,  is  assigned  to 
VN h  but  is  not  located  on  the  curve  I’/,  then  its  target  is 
set  to  the  nearest  point  on  the  curve,  nondeterministically 
choosing  one  if  there  are  several  (lines  32-33);  if  CN ,  is 
either  the  first  or  last  client  node  on  F /,  then  its  target  is  set 
to  the  corresponding  endpoint  of  Tp  (lines  35-36);  if  CN , 
is  on  the  curve  but  is  not  the  first  or  last  client  node  then 
its  target  is  moved  to  the  mid-point  of  the  locations  of  the 
preceding  and  succeeding  CNs  on  the  curve  (line  38).  For 
the  last  two  computations  a  sequence  seq  of  nodes  on  the 
curve  sorted  by  curve  location  is  used  (line  27). 

VN h  finally  broadcasts  the  new  target  waypoints  for  the 
round  through  a  target-update  message  to  its  CN s. 

VI.  Correctness  of  Algorithm 

We  say  CNi,i  £  I ,  is  active  in  round  t  if  its  mode  is 
active  for  the  duration  of  round  t.  A  VN h,  h  £  Ji,  is  active 
in  round  t  if  there  is  some  active  CN  i  with  x.(  £  Bh  for  the 
duration  of  rounds  t  —  1  and  t.  Thus,  none  of  the  VN s  is 
active  in  the  starting  round.  We  use  the  following  notation: 
In(t)  is  the  set  of  ids  h  £  H  of  VN s  that  are  active  in  round 
t  and  for  which  qh  7^  0.  Out(t )  is  the  set  of  ids  h .  £  H  of 
VNs  that  are  active  in  round  t  and  for  which  qh  =  0.  C(t) 
is  the  set  of  active  CNs  at  round  t,  and  Cin(t )  and  Cout(t) 
are  the  sets  of  active  CNs  located  in  zones  with  ids  in  In(t) 
and  Out(t),  respectively,  at  the  beginning  of  round  t. 

For  any  pair  of  neighboring  zones  Bg  and  Bh,  and  for  any 
round  t,  we  use  yg(h)(t)  to  refer  to  the  value  of  yg(h)  at 
the  point  in  time  in  round  t  when  VN g  finishes  processing 
the  first  target-trigger  message  of  round  t,  it  receives.  For 
any  /, g  £  Nbrsh  U  {ft},  in  the  absence  of  failures  and 
recoveries  of  CNs  in  round  t,  yf(h)(t)  =  yg(h)(t );  we  write 
this  simply  as  yh(t).  We  present  a  sequence  of  lemmas  that 
together  establish  the  following  theorem: 

Theorem  1:  If  there  are  no  failures  or  recoveries  of  client 
nodes  at  or  after  some  round  to,  then  within  a  finite  number 
of  rounds  after  to: 

(1)  the  set  of  CNs  assigned  to  each  VN h,  ft  £  Tt,  becomes 
fixed,  and  the  size  of  the  set  is  proportional  to  the  quantized 
length  o;,  within  a  constant  additive  term  1°(2m~1l  ancj 

(2)  all  client  nodes  in  Bh  for  which  qh  7^  0  are  located  on 
F/,  and  evenly  spaced  on  F/,  in  the  limit. 

For  the  rest  of  this  section  we  fix  a  particular  round  number 
to  and  assume  that  no  failures  or  recoveries  of  CNs  occurs 
at  or  after  round  to-  The  first  lemma  states  some  basic  facts 
about  the  assign  function  (see  Figure  7): 

Lemma  1:  In  every  round  t  >  to :  (1)  If  yp(f)  >  e  for 
some  ft  £  Tt,  then  yh(t  +  1)  >  e,  (2)  In(t)  C  In(t  +  1), 


(3)  Out(t )  C  Out(t  +  1),  (4)  Cin(t)  C  Cin(t  +  1),  and  (5) 
Cout{t  +  1)  Q  Cout(t). 

Proof:  We  fix  round  t  >  to-  (1)  From  line  6  of  the 
assign  function  (Figure  7)  it  is  clear  that  VN  h,  h  G  PL, 
reassigns  some  of  its  CN s  in  round  t  only  if  yhit)  >  e. 

(2)  For  any  VN  h,  h  G  In(t),  if  yhit)  <  e  then  VN  h 
does  assign  CN s,  and  yhit  +  1)  =  yhit),  otherwise,  from 
line  16  of  Figure  7  it  follows  that  yhit.  +  1)  >  e.  In  both 
cases  h  G  In(t  +  1). 

(3)  Same  as  (2). 

(4)  Consider  CN u  i  G  Cin(t),  such  that  CN is  assigned 
to  VN h,  h  G  In{t).  From  lines  7-11  of  Figure  7  we  see 
that  CN i  is  assigned  to  some  VN g,  g  G  Inh  U  {h}.  Since 
Irih  U  {h}  C  In{t  +  1),  the  result  follows. 

(5)  As  there  are  no  failures  and  recoveries  of  CN s,  C(t)  = 
C{t  +  1).  By  definition,  Cin(t)  U  Cout{t)  =  C(t),  Cin(t)  n 
C0ut{t)  =  0,  and  Cinit  +  1)  U  Cout{t  +  1)  =  C[t  +  1), 
Cin{t  +  1)  D  Cout{t  +1)  =0.  The  result  follows  from  part 

(4) .  ■ 


The  next  lemma  states  a  key  property  of  the  assign  func¬ 
tion  after  round  f0 :  VNg,  g  G  Out(t ),  is  never  assigned  a 
larger  number  of  CN s  in  round  t.  4- 1  than  the  largest  number 
of  CN s  that  were  assigned  to  any  of  VNg' s  neighbors  in 
round  t.  Similarly,  VN g,  g  G  In{t),  never  gets  a  density 
2/g(t+1)  of  CN s  CN s  in  round  t  +  1  that  is  greater  than  the 
highest  density  of  its  neighbors  in  round  t. 


Lemma  2:  In  every  round  t  >  to,  for  g,h  G  H  and  h  G 
Nbrsg :  (1)  If  g,h  G  Out{t),  yh(t)  =  maxfeNbrSgyf{t), 
and  yg{t)  <  yh{t ),  then  yg(t  +  1)  <  yh{t)  -  1,  and 
(2)  If  g,h  G  In(t),  =  maxfeNbrSgm( and  < 


Vh(t) 

qh 


,  then 


Vg  (*+t) 


< 


Qh 
Vh{t) 
qn 


qg 


qg  ~  qh  qtnax 

Proof:  (1)  Fix  g,  h  and  t,  as  in  the  statement  of 
the  lemma.  Since  yhit)  >  yg(t )  and  g,h  G  Out(t), 
we  see  from  line  16  of  Figure  7  that  the  number  of 
CN s  that  VNg  is  assigned  from  VN h  in  round  t  is  at 
most  p2(yh(t)  —  yg(t))/2{\lowerh(t)\  +  1).  This  is  at  most 
P2(yh(t)  -  yg(t))/ 4,  because  yh(t)  >  yg(t)  implies  that 
loiuerh{t)  >  1.  Then,  the  total  number  of  CN s  assigned 
to  VNg  in  round  t  by  all  four  of  its  neighbors  is  at  most 
P2(yh(t)-yg(t)).  Therefore,  yg(t  + 1)  <  yg (t )  +  p2 {yh (t )  - 
yg(t))  =  p2yh{t)  +  (1  -  p2)yg{t).  As  p2  <  1,  we  have 
yg{t  +  1)  <  ynit).  The  result  follows  from  integrality  of 
yg(t  +  1)  and  yhit). 

(2)  As  in  part  1,  fix  g,h  and  t.  Here  > 

Qh  Qg 

g,h  G  I  nit).  From  line  10  of  Figure  7,  it  follows  that  the 
number  of  CN s  that  VN g  is  assigned  from  VN b  in  round  t 
is  at  most  p2i^yh(t)  -  ygit))/2(\lowerhit)\  +  l).  This  is  at 
most  piC-^yhit)  —  ygit))/ 4.  Then,  the  total  number  of  CN s 
assigned  to  VN g  in  round  t  by  all  four  of  its  neighbors  is 
at  most  p2i^yh{t)  -  yg{t)).  Therefore,  yg{t  +  1)  <  (1  - 

P2)ygit)  +  p2^Vhit),  that  is  -g(^1)  <  (1  -  p2)^  + 


Mi  ^  vVA  and 


Vh  (t) 
qh 

calculation  shows  that  if  Vhl't> 

qh 


qg 


P2zff^--  AS  f2  <  We  llaVe  ^a(,|  +  1) 


7^ 


VgU) 

qg 


< 

,  then 


Vh(t) 


qg 


qh 

VhW 

qh 


.  A  simple 


Vgjt)  > 
qg  — 


The  next  lemma  states  that  there  exists  a  round  Tout  that 
is  reached  within  a  finite  number  of  rounds  after  to,  such 
that  in  every  round  t  >  Tout,  the  set  of  CN s  assigned  to 
VN h,  h  G  Outft),  does  not  change. 

Lemma  3:  There  exists  a  round  Tout  >  to  such  that  in 
any  round  t  >  Tout,  the  set  of  CN s  assigned  to  VN h,  h  G 
Outit),  is  unchanged. 

Proof:  First,  we  show  that  the  number  of  CN s  assigned 
to  VN h,  h  G  Outit),  remains  unchanged,  that  is  yhit  + 1)  = 
yhit).  Let  Novt  be  the  total  number  of  h  G  7i  such  that 
qh  =  0.  For  any  k,  1  <  k  <  Nout,  we  define  maxkit)  to 
be  the  kth  largest  number  of  CN s  that  are  assigned  to  any 
VN h,  h  G  Outit),  at  the  beginning  of  round  t  >  to- 

(max{yhit)  :  h  G  Outfi)},  if  k  =  1 

max{yh{t)  :  h  G  Outit)  A 

yhit)  <  maxk-iit)},  otherwise. 

Let  maxvnskit )  be  the  set  of  VN  ids  that  have  maxkit) 
CN s  assigned  to  them.  If  there  exists  an  l,  1  <  l  <  Nout, 
such  that  V/i  G  Out(t)  :  maxi(t.)  >  yhit),  then  for  all  k, 
l  <  k  <  N^t,  maxkit)  =  0  and  maxvnskit)  =  0. 

Let  E(t)  =  i\Coutit)\,  max\it),\maxvns\it)\, . . ., 
m(:ixNout  (f ) ,  |  rnaxvnx Nout  (f )  | )  •  Let  w  be  the  minimum 
yhit-o)  for  any  h  G  Out{to ),  and  S  =  {h  G  Out{to)  : 
yhito)  =  if}.  Observe  that  if  w  <  e,  then  Emin  = 
(iw|5|,u;,|S|,0,0...,0,0)  is  a  minimum  value  for  E)t), 
otherwise  Emin  =  (e|5|,  e,  |5|,  0,  0  . . . ,  0, 0)  is  a  minimum 
value.  It  suffices  to  show  that  for  any  round  t  >  to,  either 
E{t  +  1)  =  Eit),  that  is,  t  =  Tout,  or  E{t  +  1)  is  less 
than  Eit)  by  some  constant  amount,  meaning  there  is  a 
k.  1  <  k  <  Nout,  such  that  for  every  l,  1  <  l  <  k,  the 
Ith  component  of  E{t  +  1)  is  equal  to  the  Ith  component  of 
Eit),  and  the  kth  component  of  Eit  +  1)  is  less  than  the 
kth  component  of  Eit)  by  at  least  1. 

Consider  any  round  t  after  to-  From  Lemma  1  we  know 
that  \C outit  +  1)|  <  \Coutit)\.  If  | C out {t  +  1)|  <  \Coutit)\, 
then  the  first  component  of  E(t  + 1)  is  less  than  that  of  E{t) 
by  at  least  1.  Otherwise,  |  C out  it  +  1)|  =  |Cout(f)|.  If  for 
every  h  G  Outit),  ra  =  0  for  all  g  G  lower  hit)  (see  line  16 
of  Figure  7),  then  none  of  the  CN s  in  Coutit)  are  reassigned 
in  round  t  +  1,  and  E(t  + 1)  =  Eit).  Setting  Tout  =  t,  we  are 
done.  Otherwise,  there  exists  a  nonempty  set  of  VN s  with 
ids  in  Outit)  that  reassign  some  CN s  to  a  neighboring  VN. 
We  select  the  nonempty  set  A  of  such  VN s  with  the  highest 
number  of  assigned  CN s.  Let  A  C  maxvnskit),  for  some 
k,  1  <  k  <  Nout. 

For  any  g  G  Outit)  with  ygit)  <  max  kit),  the  maximum 
value  of  yhit)  for  any  h  G  Nbrsg  such  that  VN g  gets  some 
CN s  from  VN h  in  round  t  is  at  most  max^t).  From  Part(l) 
of  Lemma  2  it  follows  that  ygit,  +  1)  <  maxkit)  —  1. 

For  any  VN h,  h  G  A,  since  no  VN  with  y  > 
maxkit)  assigns  any  CN s  to  VN h,  yhit  +  1)  =  yhit)  — 
J2geiowerh(t)  rcls(i)'  where  rag  is  the  number  of  CN s  VN h 
assigns  to  its  neighbor  VN g  in  round  t.  We  have  shown 
above  that  for  any  g  G  Outit),  if  ygit)  <  maxk(t)  then 
yg(t  +  1)  <  maxkit)  —  1.  There  are  two  possible  cases:  (1) 


if  maxvnSk{t )  =  A,  then  the  kth  max  decreases,  maXk(t  + 
1)  <  maxk(t)  —  1.  That  is,  the  (2k  +  l)st  component  of 
E  decreases  by  at  least  1,  and  (2)  if  A  C  maxvnsk(t), 
then  maxk(t  +  1)  =  maxk(t)  and  \maxvnsk(t  +  1)|  = 
\maxvnsk(t)\  —  |A|.  That  is,  the  (2k  +  2)nd  component 
of  E  decreases  by  at  least  1.  This  implies  that  there  exists 
Tout,  such  that  the  number  of  CN s  assigned  to  each  VN  h, 
h  £  Out(t),  t  >  Tout,  remains  unchanged. 

Now  suppose  the  set  of  CNs  assigned  to  VN h  changes 
in  some  round  t  >  Tout.  Since  yh(t  +  1)  =  yh(t)  for  all 
h  G  Out(t).  Summing,  \Cout(t  +  1)|  =  \Cout(t)\  and  using 
Lemma  1  we  get  Cout(t+ 1)  =  Cout(t).  The  only  way  the  set 
of  CNs  assigned  to  VN  h  could  change,  without  changing 
yh.  and  the  set  Cout,  is  if  there  existed  a  cyclic  sequence  of 
VNs  with  ids  in  Out(t)  in  which  each  VN  gives  up  c  >  0 
CNs  to  its  successor  VN  in  the  sequence,  and  receives  c 
CNs  from  its  predecessor.  However,  such  a  cycle  of  VNs 
cannot  exist  because  the  lower  set  imposes  a  strict  partial 
ordering  on  the  VNs.  ■ 

For  the  rest  of  the  section  we  fix  Tout  to  be  the  first  round 
after  to,  at  which  the  property  stated  by  Lemma  3  holds. 
Lemma  3  implies  that  in  every  round  t  >  Tout,  In(t)  = 
In(Tout),  Out(t)  =  Out (Tout),  Cin(t)  =  Cin(Tout), 
and  C out (f)  =  Cout (Tout)',  we  denote  these  simply  as 
In,Out,Cin ,  and  Cout .  The  next  lemma  states  a  property 
similar  to  that  of  Lemma  3  for  VN  h,  h  £  In,  and  its  proof 
is  similar  to  the  proofs  of  Lemma  3,  and  uses  part  (2)  of 
Lemma  2. 

Lemma  4:  There  exists  a  round  Tstab  >  Tout  such  that  in 
every  round  t  >  Tstab,  the  set  of  CNs  assigned  to  VN  h, 
h  £  In,  is  unchanged. 

The  following  lemma  bounds  the  total  number  of  CNs 
located  in  zones  with  ids  in  Out  to  be  0(m3). 

Lemma  5:  In  every  round  t  >  Tout,  \Cout(t)\  =  0(m3). 

Proof:  From  Lemma  3,  the  set  of  CN s  assigned  to  each 
VN h,  h  £  Out(t),  is  unchanged  in  every  round  t  >  Tout- 
This  implies  that  in  any  round  t  >  Tout,  the  number  of  CN s 
assigned  by  VN h  to  any  of  its  neighbors  is  0.  Therefore, 
from  line  20  of  Figure  7,  for  any  boundary  VN g,  (yg(t)  — 
e)/\Ing\  <  1.  Ing  is  the  (constant)  set  of  h  £  Nbrsg  with 
qh  7^  0.  Since  \Ing\  <  4,  yg(t)  <  4  +  e.  From  line  16 
of  Figure  7,  for  any  non-boundary  VNg,  g  £  Out(f),  that 
is  1-hop  away  from  a  boundary  VN h,  2(\n,wel~(t)\+i)  <  L 
Since  \lowerg(t)\  <  4,  yg(t)  <  ^  +  4  +  e.  Inducting  on  the 
number  of  hops,  the  maximum  number  of  CN s  assigned  to 
a  VNg ,  g  £  Out(t),  at  l  hops  from  the  boundary  is  at  most 
—  +  e  +  4.  Since  for  any  l,  1  <  l  <  2m  —  1,  there  can  be  at 
most  m  VNs  at  (-hop  distance  from  the  boundary,  summing 
gives  \Cout\  <  (e  +  4) (2 m  -  1  )m  +  —  =  0(m3). 

U 

For  the  rest  of  the  section  we  fix  Tstab  to  be  the  first  round 
after  Tout,  at  which  the  property  stated  by  Lemma  4  holds. 
The  next  lemma  states  that  the  number  of  CN s  assigned  to 
each  VN h,  h  £  In,  in  the  stable  assignment  after  Tstab  is 


proportional  to  (p,  within  a  constant  additive  term. 
Lemma  6:  In  every  round  t  >  Tstab,  for  g,h  £  In(t): 


Vh(t)  yg(t) 

< 

'10(2  to  -  1)' 

Qh  qg 

0_minP2 

Proof:  Consider  a  pair  of  VNs  for  neighboring  zones 
Bg  and  If,,,  g,h  £  In.  Assume  w.l.o.g.  yh(t)  >  yg(t).  From 
line  10  of  Figure  7,  it  follows  that  p2(^Uh(t)  —  yg(t))  < 
2(\lowerh(t)\  +  1).  Since  \lowerh(t)\  <4,  |  |  < 

-12-  <  — — — .  By  induction  on  the  number  of  hops  from  1 
to  2m  —  1  between  any  two  VNs,  the  result  follows.  ■ 

From  line  33  of  Figure  7,  it  follows  immediately  that  by 
the  beginning  of  round  Tstab  +  2,  all  CN s  in  (7jn  are  located 
on  the  curve  T.  This  establishes  that  the  VN  algorithm 
satisfies  our  second  goal.  The  next  lemma  states  that  the 
locations  of  the  CNs  in  each  zone  If  ,,  h  £  In,  are  evenly 
spaced  on  If,  in  the  limit. 

Lemma  7:  Consider  a  sequence  of  rounds  t\  = 
Tstab,  ,tn.  As  n  — >  oo,  the  locations  of  CNs  in  Bh, 
h  £  In,  are  evenly  spaced  on  1’/,  . 

Proof:  From  Lemma  4  we  know  that  the  set  of  CNs 
assigned  to  each  VN h,  h  £  In,  remains  unchanged.  Then, 
at  the  beginning  of  round  t-2,  every  CN  assigned  to  VN h  is 
located  in  Bh  and  is  on  the  curve  If,.  Assume  w.l.o.g.  that 
VN h  is  assigned  at  least  two  CNs.  Then,  at  the  beginning 
of  round  f3,  one  CN  is  positioned  at  each  endpoint  of  I’/,, 
namely  at  Yh(inf(Ph))  and  Th(sup(Ph))-  From  lines  35- 
36  of  Figure  7,  we  see  that  the  target  points  for  these 
endpoint  CNs  are  not  changed  in  successive  rounds.  Let 
seqh(t2)  =  (po,i(o)),  ■■■,  (Pn+i,  *(n+i))»  where  yh  =  n  +  2, 
Po  =  inf(Ph),  and  pn+\  =  sup(Ph).  From  line  38  of 
Figure  7,  for  any  i,  1  <  i  <  n,  the  ith  element  in  seqh 
at  round  t±,  k  >  2,  is  given  by: 

Pi (4+1 )  =  Pi(tk)  +  Pi  I  - 2 - PiVk)  J  ■ 

For  the  endpoints,  Pi(tk+ 1)  =  Pi(tk)-  Let  the  ith  evenly 
spaced  point  on  the  curve  F/,  between  the  two  endpoints  be 
X,;.  The  parameter  value  p,  corresponding  to  x,  is  given  by 
Pi  =  Po  +  (pn+i  —  Po)-  In  what  follows,  we  show  that  as 
n  — ►  oo,  the  pt  converge  to  p,  for  every  i,  0  <  i  <  n+ 1,  that 
is,  the  location  of  the  non-endpoint  CNs  are  evenly  spaced 
on  Th.  The  rest  of  this  proof  is  exactly  the  same  as  the  proof 
of  Theorem  3  in  [8]  in  which  the  authors  prove  convergence 
of  points  on  a  straight  line  with  even  spacing. 

Observe  that pj  =  Kpi-i+Pi+i)  =  (l- pi)pi  +  Ef(pi-i  + 
Pi+i).  Define  error  at  step  k,  k  >  2,  as  et(k)  =  pt(tk)  — 
Pi.  Therefore,  for  each  i,  2  <  i  <  n  —  1,  efk  +  1)  = 
Pi(tk+ 1)  -  Pi  =  (1  -  Pi)ei(k)  +  ^(e»_i (k)  +  ei+i(k)), 
e\ (k  +  1)  =  (1  -  pi)ei(fc)  +  ^e2 (k),  and  en(k  +  1)  = 
(1— pi)en(fc)+^en_i(fc).  The  matrix  for  this  can  be  written 


as:  e(k  +  1)  =  Te(fc),  where  T  is  an  n  x  n  matrix: 


l- pi  pi/2  0  0  ...  0 

Pi/2  l-/Oi  Pi/2  0  ...  0 

0  ...  0  Pi/2  1  —  pi  pi/2 

0  ...  0  0  1  -  pi  pi/2 


Using  symmetry  of  T,  p\  <  1,  and  some  standard  theorems 
from  control  theory,  it  follows  that  the  largest  eigenvalue  of 
T  is  less  than  1.  This  implies  limk^ooTk  =  0,  which  implies 

lirrik^ooe{k)  =  0.  ■ 


VII.  Simulation  Results 

In  this  section  we  briefly  describe  the  performance  of  the 
algorithm  as  observed  in  Matlab  simulations.  In  particular 
we  study  the  stabilization  time  Tstab,  that  is,  the  number  of 
rounds  required  to  achieve  a  stable  distribution  of  PNs  over 
zones  in  TL,  for  different  types  of  target  curves.  The  inputs 
to  the  Matlab  function  that  models  the  assign  function  of 
our  algorithm  are:  (1)  a  value  of  the  parameter  e  of  the 
algorithm,  and  (2)  two  m  x  to  matrices  corresponding  to 
the  initial  and  the  target  distribution  of  PNs  respectively. 
We  have  performed  simulations  for  different  values  of  m 
and  for  different  target  distributions.  In  all  the  experiments, 
we  fix  the  value  of  e  to  be  5,  the  number  of  participating 
PNs  to  be  105,  and  the  initial  locations  of  all  PNs  to  be  the 
lower  right  corner  of  B. 

The  bar  charts  in  in  Figure  8  show  the  distribution  of 
the  PNs  (dark  bars)  at  different  stages  in  a  typical  run  of 
the  algorithm,  and  the  target  distribution  (light  bars).  From 
top  to  bottom,  the  charts  show  the  distribution  of  the  PNs 
after  the  first  round,  at  the  end  of  round  Tout,  and  the  stable 
distribution  attained  at  round  Tstob,  respectively. 

The  plots  in  Figure  9  show  the  values  of  Tout  and  Tstat, 
for  values  of  m  ranging  from  10  to  20,  and  for  three  different 
curves.  Cl  covers  a  rectangular  region  at  the  lower-right 
corner  of  B,  C2  covers  a  rectangular  region  at  the  top-left 
corner  of  B,  and  C3  is  an  annular  ring  at  the  center  of  B. 
From  these  plots  we  observe  that  if  the  curve  is  located  far 
from  the  initial  position  of  the  PNs,  as  in  the  case  of  C2, 
then  the  stabilization  time  Tstab  is  dominated  by  Tout. 


VIII.  Implementing  the  Virtual  Node  Layer 

In  addition  to  client  CN ,,  a  physical  node  IN,,  i  £  I,  in 
zone  Bh  runs  a  TOBcasti^h  service  and  a  VNEith,  h  £  H, 
algorithm  (see  Figure  10)  to  help  implement  each  virtual 
node  VN h  and  the  VLBcast  service  of  the  virtual  layer. 

In  this  section  we  present  a  sketch  of  our  implementation 
of  the  virtual  layer  by  the  physical  layer.  Our  implementation 
is  an  adaptation  of  techniques  from  [4]  to  emulate  a  virtual 
mobile  node.  The  only  substantive  changes  made  in  our 
current  implementation  are:  (1)  the  changing  of  virtual  node 
locations  to  be  stationary,  (2)  the  replacement  of  a  periodic 
location  update  with  a  continuous  real-time  location  update, 
and  (3)  the  restart  of  a  virtual  node  as  soon  as  a  physical 
node  discovers  it  is  in  a  failed  virtual  node’s  zone.  The 
virtual  nodes  we  implement  here  are  also  modeled  differently 


Fig.  8.  Simulation  results  show  the  actual  (dark)  and  target  (light) 
distribution  of  PNs  over  Pi,  at  the  end  of  various  rounds:  the  first  round 
(top),  at  Tout  (middle),  and  at  Tstafo  (bottom). 


Fig.  9.  Plot  of  Tout  and  Tsta 5  for  three  different  kinds  of  curves  with 
varying  length  m  of  each  side  of  the  bounded  plane  B.  Cl  covers  a 
rectangular  region  at  the  lower-right  comer  of  B,  C2  covers  a  rectangular 
region  at  the  top-left  comer  of  B,  and  C3  is  an  annular  ring  at  the  center 
of  B. 


Fig.  10.  PNi's  subautomata:  A  physical  node  runs  several  programs, 
including  VNE  and  TO  Beast  automata  as  well  as  a  CN  automaton. 


from  those  in  [4],  as  MMT  automata,  rather  than  simple  I/O 
automata. 

We  use  a  standard  replicated  state  machine  approach  to 
implement  robust  virtual  nodes  that  takes  advantage  of  a 
TOBcast  service  to  ensure  that  all  VNE s  in  a  zone  receive 
the  same  messages  in  the  same  order.  Using  the  LBcast 
service  of  the  physical  nodes  and  common  knowledge  about 
realtime,  the  totally  ordered  broadcast  service  TOBcast 
for  a  zone  can  be  implemented  as  follows:  At  the  time  of 
sending,  a  message  is  tagged  with  the  sender’s  identifier, 
zone  id,  a  sequence  number,  and  a  timestamp,  which  is  the 
current  value  of  realtime.  The  tags  define  a  total  order  on 
sent  messages,  used  in  delivery.  Before  delivering  a  message 


TOBcasti'h  waits  until  dp  +  e  time  has  elapsed  since  it 
was  sent,  ensuring  that  earlier  messages  were  received.  It 
then  delivers  the  messages  for  the  timestamp  in  order  of 
sender  id  and  sequence  number.  (As  a  technical  detail,  we 
actually  order  all  join-req  messages  (described  shortly)  after 
any  other  messages  for  a  particular  time.)  TOBcastih  only 
processes  messages  tagged  for  zone  Bh- 

Each  VNEi^h  independently  maintains  the  state  of  VN  h 
and  simulates  performing  actions  of  the  VN  on  that  state.  In 
order  to  keep  the  state  replication  consistent  across  different 
VNEs  running  on  different  physical  nodes  in  the  same 
zone,  when  VNEjj,  wants  to  simulate  an  action  of  the  VN 
(such  as  that  of  receiving  a  message  at  the  VN  that  was 
actually  received  by  VNEi ^),  it  broadcasts  a  suggestion  to 
perform  the  action  to  the  other  VNEs  of  the  zone  using  the 
TOBcast  service.  When  an  action  suggestion  is  received  by 
VNEi^h,  it  is  saved  in  a  pending-action  queue.  Actions  are 
removed  from  a  pending-action  queue  in  order  by  VNEi^h 
and  simulated  on  VNE^h's  local  version  of  the  VN  state. 
A  completed  action  is  then  moved  into  a  completed- action 
queue,  referenced  by  VNEi^h  to  prevent  reprocessing  of 
completed  actions. 

When  a  VNE  enters  a  zone,  it  executes  a  join  protocol  to 
get  the  zone’s  VN  state.  The  join  protocol  begins  by  using 
TOBcast  to  send  a  join-req  message.  Whenever  a  VNE 
receives  its  own  join-req  message,  it  starts  saving  messages 
to  process  in  its  pending-action  queue.  If  a  VNE  that  has 
already  joined  receives  the  join-req,  it  uses  TOBcast  to  send 
a  join-ack  containing  a  copy  of  its  version  of  the  VN  state. 
When  the  joining  VNE  receives  the  join-ack,  it  copies  the 
included  VN  state  and  starts  processing  the  actions  in  its 
pending-action  queue.  If  a  VNE’ s  join-req  is  not  answered 
in  3 dp  +  3e  time,  indicating  the  VN  is  failed,  the  VNE  will 
reset  the  VN  e  time  later  by  using  TOBcast  to  send  a  reset 
message.  When  a  VNE  receives  a  reset  message,  it  sets  the 
VN  state  to  its  initial  state,  clears  the  pending-action  queue, 
and  starts  simulating  the  VN. 

Theorem  2:  Assuming  Rp  >  v/5 b,  the  TOBcastith, 
VNEith,  i  £  T,h  £  TL,  and  trivial  client  implementation 
correctly  implement  the  Virtual  Node  abstraction  with  VN 
task  upper  time  bound  cImmt  =  2 dp  +  2e,  VTV-startup  time 
dr  =  Adp  +  5e,  VLBcast  broadcast  radius  Rv  >  b,  and 
VLBcast  maximum  message  delay  dv  =  2 dp  +  e. 

Proof:  The  correctness  of  the  implementation  of  the 
Virtual  Node  layer  largely  follows  from  the  proof  of  cor¬ 
rectness  for  the  implementation  of  the  VMN  layer  in  [4]. 
We  here  discuss  the  correctness  of  the  implementation  with 
respect  to:  (1)  the  task  upper  bound,  (2)  the  VTV-startup  time, 
and  (3)  the  requirements  for  LBcast  and  VLBcast. 

(1)  Once  one  of  an  abstract  VNfs  output  or  internal  tran¬ 
sitions  is  enabled,  the  precondition  for  sending  a  suggestion 
to  simulate  the  action  through  TOBcast  is  satisfied  at  all 
VNEij-,  for  PN i  in  Bh,  and  the  broadcast  occurs.  It  takes 
at  most  dp  +  e  time  for  the  message  to  be  delivered  at  other 
VNEi^h  for  PN i  in  If,,  after  which  the  action  is  simulated. 
However,  it  is  possible  for  all  active  VNEs  to  fail  right  after 


sending  a  join-ack  to  a  new  VNE  and  before  proposing 
an  enabled  action,  leaving  the  new  VNE  to  broadcast  the 
simulation  proposal  dp+e  later,  when  it  receives  the  join-ack. 
Given  that  PN  transitions  are  assumed  to  be  instantaneous, 

d-MMT  =  2dp  +  2e. 

(2)  If  PN i  enters  a  zone  If,  with  a  failed  VN,  its 
VNEi  h’s  join-req  will  not  be  answered  in  3 dp  +  3e  time, 
and  the  VNE  will  send  a  reset  message  an  additional  e  later. 
It  takes  the  VNE  at  most  dp  +  e  time  to  receive  the  reset 
message  and  restart  the  VN.  The  total  time  4 dp  +  5e  for  a 
joining  node  to  succeed  in  restarting  a  VN  is  dr. 

(3)  As  in  [4],  dv  =  2 dp  +  e  since  the  underlying  LBcast 

service  used  to  implement  VLBcast  takes  up  to  dp  time 
to  deliver  a  transmitted  message  from  a  VN  or  CN ,  after 
which  TOBcast  takes  an  additional  dp+e  time  to  redeliver  a 
message  at  a  receiving  VN.  Also  similarly  to  [4],  we  require 
that  Rp  >  y/b  b,  in  order  to  guarantee  that  Rv  >  b ,  allowing 
a  CN t  in  Bh ,  i  €  X,  h  G  H,  and  VN  h  to  communicate,  and 
a  VN h  (located  at  o h)  and  each  of  its  neighboring  zones’ 
VN g,  g  €  Nbrs(h),  (located  at  og)  to  communicate.  This  is 
because  a  VNE  emulating  a  zone  Bh  can  be  as  far  away  as 
\J (2b)2  +  b2  from  a  VNE  emulating  the  VN  of  neighboring 
zone  Bg.  To  guarantee  the  two  can  communicate  while 
emulating  their  respective  VN s,  the  broadcast  radius  Rp  of 
the  physical  LBcast  service  must  be  be  at  least  \/B b.  Unlike 

[4],  however,  we  do  not  require  an  additional  tolerance  factor 
to  account  for  periodic  location  updates  from  the  RW\  here, 
the  RW  automaton  is  assumed  to  continually  update  the 
VNE  of  its  current  location.  ■ 

IX.  Future  work  and  extensions 

We  believe  the  framework  introduced  in  this  paper  can 
be  useful  in  simplifying  the  coordination  of  mobile  nodes 
to  solve  a  variety  of  other,  more  complex,  problems.  One 
promising  avenue  of  future  work  would  be  to  employ  our 
framework  for  some  of  those  problems.  As  one  example,  in 
the  control  algorithm  presented  in  this  paper,  each  virtual 
node  VN  uses  only  local  information  about  the  target  curve 
r.  This  use  of  only  local  information  should  adapt  well  to  a 
problem  extension  where  the  curve  is  dynamically  changing. 
The  curve  (or  point,  even)  could  be  moving  targets  being 
tracked.  In  this  case,  the  framework  for  coordination  of  nodes 
we  present  here  is  useful  for  two  reasons:  (1)  maintaining 
alive  VN s  to  detect  targets  and  (2)  guiding  physical  nodes 
to  the  moving  targets. 

Also  of  interest  is  an  in-depth  analysis  of  the  stability 
of  the  control  algorithm  employed  in  this  paper  in  the 
face  of  some  regular  rate  of  physical  node  addition  and 
removal.  While  there  has  been  a  paucity  of  work  in  this 
area,  such  analyses  are  extremely  important  in  the  evaluation 
of  solutions  to  most  any  control  algorithms  for  dynamic 
systems. 
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Appendix 


Signature: 

Input 

receive(ffi),  /j,  m  a  client  message 
TOBcast-rcv(m)i^,  m  a  TOBcast  message 

Output 

send (m)i  h,  m  a  client  message 
TOBcast(m);  /,  rn  a  TOBcast  message 

Internal 

zone-update;, h 
joini>h 
restart,, h 

init-actionfacf);  /!,  act  G  VN^-sig  \  inputs 
simulate-action(ac/)ii/l,  act  g  VNh.sig 

ack-join;^ 

Variables: 

Input 

x;  G  B,  current  location  of  mobile  node 
realtime  G  /V'  (l 

Internal 

status  G  {joining,  listening,  active},  initially  active 
h  G  HU  jl},  zone  id,  initially  1. 
val  G  VN i,. stales,  state  of  VW/,.  initially  V N^. start 
pending-join,  max  id  of  pending  join  reqs,  initially  0 
completed -join,  max  id  of  answered  join  reqs,  initially  0 
join-id,  time  of  join-req,  initially  0 

pending -actions,  queue  of  VN^.  actions  to  be  simulated,  initially  0 
completed -actions,  queue  of  V  N^. actions  simulated,  initially  0 
TOBcast-out,  queue  of  outgoing  TOBcast  msgs,  initially  0 
local -out,  queue  of  outgoing  client  messages,  initially  0 

Trajectories: 

Stop  when  any  Precondition  is  satisfied 


Fig.  1 1 .  Signature,  variables,  trajectories  of  VNEi  /,  algorithm  imple¬ 
menting  VN  n- 


Input  receive(m)i  ^ 

Effect 

TOBcast-out  <—  TOBcast-out  U  {(simulate,  (receive,  m),  ±)} 

Output  send(m); 

Precondition 

local-out  /  0  A  m  =  head  (local-out) 

Effect 

local-out  <—  tail  (local-out) 

Internal  init-actionjacf);,/. 

Precondition 

status  =  active  A  x  G  Bf,  A  S (val,  act)  /  _L 

Effect 

TOBcast-out  <—  TOBcast-out  U  {(simulate,  act,  ( realtime ,  ij) } 

Internal  join;,/. 

Precondition 

status  =  idle  A  x  G  Bh 
Effect 

status  <—  joining 
join -id  <—  realtime 

TOBcast-out  <—  TOBcast-out  U  {(join-req,  _L ,  join-id)} 

Internal  restart,.;, 

Precondition 

status  =  listening  Ax  G  Bit  A  realtime  =  join-id  +  3  dv  +  4e 

Effect 

TOBcast-out  <—  TOBcast-out  U  {(reset)} 

Internal  zone-update;,/. 

Precondition 
x  £  Bh 
Effect 

status  idle 

h  id  of  zone  h/  such  that  x  e  Bh, 
va/  VNh  -start 

pending -actions  0 

Internal  simulate-actionfact)^ 

Precondition 

status  =  active  Ax  G  A  head  {pending -actions)  =  (simulate,  acr,  o/d) 
Effect 

dequeue  (pending -actions) 

if  ((simulate,  act,  oid )  £  completed -actions  A  6 (val,  act )  ^  _L)  then 
val  8  (val,  act) 
if  act  =  (send,  m)  then 

local-out  local-out  U  {m} 

completed -actions  < —  completed -actions  U  {(simulate,  act,  o/d) } 


Internal  ack-join^ 

Precondition 

status  =  active  Ax  £  Bh  A  pending-join  >  completed -join 
pending -actions  =  0  A  Vac?  £  VNh.sig  \  inputs:  8  (val,  act)=  _L 

Effect 

TOBcast-out  *—  TOBcast-out  U  {(join-ack,(va/,  completed -actions),  pending -join)} 
completed -join  pending -join 


Input  TO  Beast  -  rev  ( (optype ,  param,  oid))i  h 

Effect 

if  optype  =  simulate  then 

if  status  =  listening  or  active  then 

enqueue  (pending -actions,  (simulate,  param,  oid)) 
if  optype  =  join-req  then 

pending -join  <—  mnx(pending  -join,  oid) 
if  (status  =  joining  A  oid  =  join -id)  then 
status  <—  listening 
if  optype  =  join-ack  then 

completed -join  msix(completed-join,  oid) 
if  (status  =  listening  and  oid  >  join -id)  then 
status  active 

(val,  completed -actions)  param 

if  optype  =  reset  then 
status  active 
pending -actions  0 


Output  TOBcast(ra)^h 
Precondition 

TOBcast-out  /  0  Am  =  head  (7D.Bca.sf -car) 

Effect 

TOBcast-out  tail(7DBca.st-aM?) 


Fig.  12.  Transitions  of  VNEi^h  algorithm. 


